Audio to Indian Sign Language Translator

Abstract

Introduction

I. INTRODUCTION

People who are deaf or hard of hearing often rely on sign language as their primary mode of communication because it is widely used across the globe. Despite the fact that many countries utilise computers, relatively little research has been done to identify ISL. The fundamental architecture of the vast majority of systems is based on: Translation techniques such as Direct Translation, Transfer-based Architecture, and Statistical Machine Translation The system has been implemented. The focus is primarily on the Indian Sign Language. According to the census completed in 2011, there are 63 million people in India who have hearing problems, which accounts for 6.3 percent of the total population. It is estimated that between 76 and 89 percent of hearing-impaired Indians have no knowledge of the language in either of its signed, spoken, or written forms. The low literacy rate can be attributed to a number of problems, including the absence of sign language interpreters, a dearth of Indian Sign Language equipment, and a dearth of research on the subject of Indian Sign Language.

The difficulties that are experienced by the deaf population in India are the impetus for the creation of the system that we have recommended. This system is designed to perform a sign language translation for each individual word that is taken in as input. A video that displays the sign language interpretation of the spoken sentence is accompanied by a voice recognizer that turns spoken sentences into expressions and silences and recognises it as text - a sequence of words. This system is comprised of both of these components. It is a flexible method that is capable of considerably cutting down on the amount of work and the parallel corpus that are required for converting a speech to sign language translation system to a new area.

A.  Problem Statement

A language known as sign language is one in which information is conveyed via the use of manual communication methods such as facial expressions, hand gestures, and other bodily motions. In order to accomplish the task of translating written language into sign language, this project utilises movies that are dedicated to specific terms.

B.  Proposed System

This proposed system is developed to overcome the challenges faced by the deaf people. Basically, this system is planned to translate words that is accepted as input to sign language. This model translates the words based on Indian Sign Language.

1. Natural Language Processing: Some unwanted words such as ‘are’, ‘as’, ‘is’, ‘was’, interjections, commas, symbols, non-root verbs etc. Hence, this system takes away those stop words from the processed speech.
2. Root Words: The words may be in gerund form, plural form, or adjective form. The proposed system will remove these forms of the words and find the root word from those words. These root words will be helpful in the effective conversion into sign language.
3. Dataset: The system has a large dataset of Indian sign language words to map according to the text or text recognized from the speech. So, it will be helpful to all deaf people in India. It makes the people understand most of the speech or text.

II. LITERATURE SURVEY

As we know, India is a multilingual country. There are different state and all its state have their own regional language and al its regional language have its own sign language. In India there is less research have done on this project.

The ISLRTC (Indian Sign Language Research & Training Center) have taken step to look on this type of application. For deaf people in India provides practices to learn and understand American Sign Language ASL because it is very easy to learn and understand and after this learn sign language. Approx. 98% of 1.1, million deaf people in India are illiterate. Deaf schools are trying to implement this strategy with hearing aids, but this is not possible to apply on community. Research have tries to find a new solution to this root problem. The strategy for producing gloss aligned with ISL grammar. The primary aim of this project is transformation from audio to sign language is formation of phrases that help to identify the accurate American Sign Language ASL grammar phrases. Each sentence from the input is converted into tokens, and after that each tokens particular actions are tokens (Like each word are convert in its root word and helping words are removed). All those word from input which are not related to ASL Gloss are removed. After this processing of input proper noun is outputted and separated by hyphen and remaining words are also give output by applying various rules.

For example-INDIA, this alphabet of proper noun is I-N-D-I-A corresponding to the use of conventional sign language. This system can cause destructive nature is only human error found. Gloss storing can be one of the reasons of human error.

As the second most prevalent cause of disability, Neha Poddar, Shrushti Rao, Shruti Sawant, Vrushali Somavanshi, and Professor Sumita Chandak noted in their paper that the prevalence of deafness in India is fairly high. A portable interpreting device that converts higher mathematics sign language into corresponding text and voice could be of great assistance to the deaf and solve numerous problems.

III. METHODOLOGY

A. Forms of Input

Our project is intended to get inputs two formats. The inputs can be of forms:

1. Text input
2. Live speech input

B.. Speech Recognition

The microphone in our system is where the live speech comes in. The Python package PyAudio is used to do this. PyAudio is a package for Python that lets you record audio on many different platforms. Google Cloud Speech is then used to change the audio into text.

C. Pre-processing of Text

In the part for providing input, sentences are broken up into tokens. Each sentence is given a classification based on the results of machine learning and natural language processing tools. This module's goal is to transform input text that uses English grammar into material that uses the grammar of International Sign Language. Following the tokenization step, the next  step is the POS tagging, which may be defined as the process of matching each word with its respective part of speech. Also known as point-of-sale (POS) tagging in popular parlance. Included in the category known as Part of Speech are conjunctions, adjectives, verbs, adverbs, nouns, pronouns, and adverbial phrases. After the POS tagging has been finished, It will then be possible for us to continue the process of removing all of the stop words. Simply put, "stop words" are words that are used in language that do not contribute significantly to the overall meaning of the sentence. They do not contribute significantly to the meaning of the statement; thus they can be eliminated and the meaning will remain the same. In addition to this, once we have removed them, we will employ a process called lemmatization to get rid of the inflectional ending and return the word to its base form, which is called the lemma. It will return either saw or see depending on whether the token saw was used as a noun or a verb in the sentence while it was being lemmatized.

D. Result

Output from the ISL conversion phase of the input sentence is passed to the video generation phase, wherein each of the words in the sentence is looked up in the database for its corresponding video file. Then these files are all concatenated to produce a more structured, informative, and easy-to-understand visual depiction of Indian Sign Language.

The final product of this unit will be a video clip containing text translated into ISL. The database will have a video for each individual word, and the final video will be a compilation of all these words into a single video.

IV. FLOW CHART

V. RECOMMENDATIONS FOR FUTURE STUDIES

In light of the fact that the ISL is still in its infancy and that very little research has been conducted on the subject, the dictionary could benefit from the addition of a great number of fresh videos explaining the meanings of various words. This would broaden the ISL's application and make it easier for people to communicate with one another using this language. Users will be able to convert text to Indian Sign Language by manually entering the input sentence after the text-to-speech integration that can be performed in the project to enable better communication strategies has been completed. Moreover, text to speech integration can be performed in the project to enable better communication strategies.

Conclusion

Here, we\'ve tried to make a model that will let people with disabilities express themselves in their own way, which will make it easier for them to fit in with the rest of the world. The audio we give our proposed model will be successfully turned into an animation. As the ISL Dictionary grows, there will be many ways to improve this route. The number of words in the ISL is small, so adding new words to their dictionary to make it bigger would help in many ways. Text-and-speech integration can also be done on a project to make it easier for people to talk to each other. This will let users turn text into Indian Sign Language by typing it in by hand.

References

[1] F. Shaikh, S. Darunde, N. Wahie, and S. Mali, \"Sign Language Translation System for Railway Station Announcements,\" 2019 IEEE Bombay Section Signature Conference (IBSSC), 2019, pp. 1-6, DOI:10.1109/IBSSC47189.2019.8973041. [2] K. Saija, S. Sangeetha and V. Shah, \"WordNet Based Sign Language Machine Translation: from English Voice to ISL Gloss,\" 2019 IEEE 16th India Council International Conference (INDICON), 2019, pp. 1-4, DOI: 10.1109/INDICON47234.2019.9029074. [3] B.D. Patel, H. B. Patel, M. A. Khanvilkar, N. R. Patel and T. Akilan, \"ES2ISL: An Advancement in Speech to Sign Language Translation using 3D Avatar Animator,\" 2020 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), 2020, pp. 1-5, DOI: 10.1109/CCECE47787.2020.9255783. [4] Sunitha K. A, Anitha Saraswathi.P, Aarthi.M, Jayapriya. K, Lingam Sunny, “Deaf Mute Communication Interpreter- A Review”, International Journal of Applied Engineering Research, Volume 11, pp 290-296, 2016. [5] Mathavan Suresh Anand, Nagarajan Mohan Kumar, Angappan Kumare- san, “An Efficient Framework for Indian Sign Language Recognition Using Wavelet Transform” Circuits and Systems, Volume 7, pp 1874- 1883, 2016. [6] Chandandeep Kaur, Nivit Gill, “An Automated System for Indian Sign Language Recognition”, International Journal of Advanced Research in Computer Science and Software Engineering.

Copyright

Copyright © 2022 Harshita Mishra, Mansi Sharma, Muskan Ali Qureshi, Shivani Chaudhary. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.